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Abstract. Quantitative extensions of temporal logics have recently at¬ 
tracted significant attention. In this work, we study frequency LTL (fLTL), 
an extension of LTL which allows to speak about frequencies of events 
along an execution. Such an extension is particularly useful for proba¬ 
bilistic systems that often cannot fulfil strict qualitative guarantees on 
the behaviour. It has been recently shown that controller synthesis for 
Markov decision processes and fLTL is decidable when all the bounds 
on frequencies are 1. As a step towards a complete quantitative solution, 
we show that the problem is decidable for the fragment fLTLycu, where 
U does not occur in the scope of G (but still F can). Our solution is 
based on a novel translation of such quantitative formulae into equivalent 
deterministic automata. 


1 Introduction 

Markov decision processes (MDP) are a common choice when modelling systems 
that exhibit (un)controllable and probabilistic behaviour. In controller synthesis 
of MDPs, the goal is then to steer the system so that it meets certain property. 
Many properties specifying the desired behaviour, such as “the system is always 
responsive” can be easily captured by Linear Temporal Logic (LTL). This logic 
is in its nature qualitative and cannot express quantitative linear-time proper¬ 
ties such as “a given failure happens only rarelij'. To overcome this limitation, 
especially apparent for stochastic systems, extensions of LTL with frequency 
operators have been recently studied m- 

Such extensions come at a cost, and for example the “frequency until” op¬ 
erator can make the controller-synthesis problem undecidable already for non¬ 
stochastic systems m- It turns out |19I30I31] that a way of providing signif¬ 
icant added expressive power while preserving tractability is to extend LTL 
only by the “frequency globally” formulae Such a formula is satisfied 

if the long-run frequency of satisfying ip on an infinite path is at least p. More 
formally, G-^ip is true on an infinite path sqSi • • • of an MDP if and only if 
^ ■\{i \ i < n and SiSi+i ■ ■ ■ satisfies (^}| is at least p as n tends to infinity. Be¬ 
cause the relevant limit might not be defined, we need to consider two distinct 
operators, and G^^, whose definitions use limit inferior and limit superior, 
respectively. We call the resulting logic frequency LTL (fLTL). 




So far, MDP controller synthesis for fLTL has been shown decidable for the 
fragment containing only the operator [TH]. Our paper makes a significant 
further step towards the ultimate goal of a model checking procedure for the 
whole fLTL. We address the general quantitative setting with arbitrary frequency 
bounds p and consider the fragment fLTL\GU) which is obtained from frequency 
LTL by preventing the U operator from occurring inside G or G-^ formulas (but 
still allowing the F operator to occur anywhere in the formula). The approach we 
take is completely different from m where ad hoc product MDP construction is 
used, heavily relying on existence of certain types of strategies in the G,^f case. 
In this paper we provide, to the best of our knowledge, the first translation of a 
quantitative logic to equivalent deterministic automata. This allows us to take 
the standard automata-theoretic approach to verification |33j : after obtaining 
the finite automaton, we do not deal with the structure of the formula originally 
given, and we solve a (reasonably simple) synthesis problem on a product of the 
single automaton with the MDP. 

Relations of various kinds of logics and automata are widely studied (see 
e.g. |32l21jllb| i. and our results provide new insights into this area for quanti¬ 
tative logics. Previous work m offered only translation of a similar logic to 
non-deterministic “mean-payoff Biichi automata” noting that it is difficult to 
give an analogous reduction to deterministic “mean-payoff Rabin automata”. 
The reason is that the non-determinism is inherently present in the form of 
guessing whether the subformulas of G-^ are satisfied on a suffix. Our construc¬ 
tion overcomes this difficulty and offers equivalent deterministic automata. It is 
a first and highly non-trivial step towards providing a reduction for the complete 
logic. 

Although our algorithm does not allow us to handle the extension of the 
whole LTL, the considered fragment fLTL\GU contains a large class of formulas 
and offers significant expressive power. It subsumes the GR(I) fragment of LTL 
[5, which has found use in synthesis for hardware designs. The U operator, 
although not allowed within a scope of a G operator, can still be used for example 
to distinguish paths based on their prefixes. As an example synthesis problem 
expressible in this fragment, consider a cluster of servers where each server plays 
either a role of a load-balancer or a worker. On startup, each server listens 
for a message specifying its role. A load-balancer forwards each request and 
only waits for a confirmation whereas a worker processes the requests itself. A 
specification for a single server in the cluster can require, for example, that the 
following formula (with propositions explained above) holds with probability at 
least 0.95: 

((ZU6) ^ X(/AFc))) A ((/Uw) ^ ^ (XpVXXp))) 

Related work. Frequency LTL was studied in another variant in m where 
a frequency until operator is introduced in two different LTL-like logics, and 
undecidability is proved for problems relevant to our setting. The work [7] also 
yields decidability with restricted nesting of the frequency until operator; as the 
decidable fragment in [7] does not contain frequency-globally operator, it is not 
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possible to express many useful properties expressible in our logic. A logic that 
speaks about frequencies on a finite interval was introduced in [30] , but the paper 
provides algorithms only for Markov chains and a bounded fragment of the logic. 

Model checking MDPs against LTL objectives relies on the automata-theoretic 
approach, namely on translating LTL to automata that are to some extent de¬ 
terministic m- This typically involves translating LTL to non-deterministic 
automata, which are then determinized using e.g. Safra’s construction. During 
the determinization, the original structure of the formula is lost, which prevents 
us from extending this technique to the frequency setting. However, an alterna¬ 
tive technique of translating LTL directly to deterministic automata has been 
developed naMni, where the logical structure is preserved. In our work, we 
extend the algorithm for LTL\gu partially sketched in [24]. In Section |6l we 
explain why adapting the algorithm for full LTL m is difficult. Translation of 
LTL\gu to other kinds of automata has been considered also in [2Ij . 

Our technique relies on a solution of a multi-objective mean-payoff problem 
on MDP mM- Previous results only consider limit inferior rewards, and so we 
cannot use them as off-the-shelf results, but need to adapt them first to our set¬ 
ting with both inferior and superior limits together with Rabin condition. There 
are several works that combine mean-payoff objectives with e.g. logics or par¬ 
ity objectives, but in most cases only simple atomic propositions can be used 
to define the payoff mm- The work [3] extends LTL with another form of 
quantitative operators, allowing accumulated weight constraint expressed using 
automata, again not allowing quantification over complex formulas. Further, [1] 
introduces a variant of LTL with a discounted-future operator. Finally, tech¬ 
niques closely related to the ones in this paper are used in |I8I14I27| . 

Our contributions. To our best knowledge, this paper gives the first decid¬ 
ability result for probabilistic verification against linear-time temporal logics ex¬ 
tended by quantitative frequency operators with complex nested subformulas of 
the logic. It works in two steps, keeping the same time complexity as for ordinary 
LTL. In the first step, a fLTL\GU formula gets translated to an equivalent de¬ 
terministic generalized Rabin automaton extended with mean-payoff objectives. 
This step is inspired by previous work |24j . but the extension with auxiliary au¬ 
tomata for G-^’ requires a different construction. The second step is the analysis 
of MDPs against conjunction of limit inferior mean-payoff, limit superior mean- 
payoff, and generalized Rabin objectives. This result is obtained by adapting 
and combining several existing involved proof techniques mm- 

The paper is organised as follows: the main algorithm is explained in Sec- 
tion[3l relegating the details of the two technical steps above to Sections [4] and [5] 

2 Preliminaries 

We use N and Q to denote the sets of non-negative integers and rational numbers. 
The set of all distributions over a countable set X is denoted by Dist{X). For a 
predicate P, the indicator function 1 p equals I if P is true, and 0 if P is false. 
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Markov decision processes (MDPs). An MDP is a tuple M = {S, A, Act, S, s) 
where S' is a finite set of states, A is a finite set of actions, Act : S ^ 2^ \ {0} 
assigns to each state s the set Act{s) of actions enabled in s, <5 : A —>• Dist{S) is a 
probabilistic transition function that given an action a gives a probability distri¬ 
bution over the successor states, and s is the initial state. To simplify notation, 
w.l.o.g. we require that every action is enabled in exactly one state. 

Strategies. A strategy in an MDP M is a “recipe” to choose actions. Formally, 
it is a function a : {SA)*S —> Dist{A) that given a finite path w, representing 
the history of a play, gives a probability distribution over the actions enabled 
in the last state. A strategy cr in M induces a Markov chain which is a 
tuple (L, P, s) where the set of locations L = {S x A)* x S encodes the history 
of the play, s is an initial location, and P is a probabilistic transition function 
that assigns to each location a probability distribution over successor locations 
defined by P(/i)(/ias) = (T(/i)(a) ■ (5(a)(s). for all/i S («S'A)*S', a G A and s G S'. 

The probability space of the runs of the Markov chain is denoted by and 
defined in the standard way | 20 ] : for reader’s convenience the construction is 
recalled in Appendix [C] 

End components. A tuple (T, B) with ^ ^ T C S and B C Act(t) is an 
end component of M if (1) for all a G B, whenever 6{a){s') > 0 then s' GT', and 
(2) for all s,t G T there is a path w = siOi • • • ak-iSk such that si = s, Sk = t, 
and all states and actions that appear in w belong to T and B, respectively. An 
end component (T, B) is a maximal end component (MEC) if it is maximal with 
respect to the componentwise subset ordering. Given an MDP, the set of MECs 
is denoted by MEC. Finally, an MDP is strongly connected if (S, A) is a MEC. 

Frequency linear temporal logic (fLTL). The formulae of the logic fLTL 
are given by the following syntax: 

(p ::= tt|ff|a|-'a|(/3A(/3|(/7V(^| X-p \ Fp \ Gp \ pXJp \ G'^p 

over a finite set Ap of atomic propositions, ixi G {>,>}, p G [0,1] Cl Q, and 
ext G {inf, sup}. A formula that is neither a conjunction, nor a disjunction is 
called non-Boolean. The set of non-Boolean subformulas of p is denoted by sf(i^). 
Words and fLTL Semantics. Let w G ( 2 ^P)“ be an infinite word. The Ah 
letter of w is denoted ic[i], i.e. w = w[0]i(;[l] • • •. We write for the finite 
word ui[f]'u;[i -|- 1] • • • w[j], and or just ic* for the suffix ui[f]'u;[* -I- 1] • • •. The 
semantics of a formula on a word w is defined inductively: for tt, ff, A, V, and 
for atomic propositions and their negations, the definition is straightforward, for 
the remaining operators we define: 

3k gN : \= Ip and 

'^0<j<k: w^\=p 

(ilpO \^Lp ‘ * ‘ ) P 

By L((p) we denote the set 

w € ( 2 ^P)“ I w \= p\ oi words satisfying p. 


w \= Xp 
w \= Fp 
w 1= Gi^ 


\= p 

3k gN : \= p 


w 1= pGip 
Vk gN : \= p w \= G'^p 


where we set lrext(gig 2 • • •) “ Hmexti^oo 7 Qi- 
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The fLTL\Qu fragment of fLTL is defined by disallowing occurrences of U 
in G-formulae, i.e. it is given by the following syntax for ip\ 

ip ::=a \ \ p ^ p \ p\J p \ X.p \ p\5p | | | G^?^ 

e ::=a | -a U A ^ U V ^ | | | G^ | G^x^^ 

Note that restricting negations to atomic propositions is without loss of gener¬ 
ality as all operators are closed under negation, for example ^G^^p = Gf^~P^p 
or Furthermore, we could easily allow cxi to range also over 

< and < as Gp^^p = G^^-P^p and G<> = Gf^-P^p. 

Automata. Let us fix a finite alphabet S. A deterministic labelled transition 
system (LTS) over A is a tuple iQ,qo,S) where Q is a finite set of states, qo is 
the initial state, and 6 : Q x S ^ Q is a partial transition function. We denote 
d{q,a) = q' also by q —> q'. A run of the LTS S over an infinite word w is a 
sequence of states S{w) = qoqi • ■ • such that <7^+1 = 5{qi,w[i]). For a finite word 
w of length n, we denote by S{w) the state qn in which S is after reading w. 

An acceptance eondition is a positive boolean formula over formal variables 

{Inf{S),Fin{S),MP'^J^{r) \ S'CQ, exte{inf, sup}, txi€{>, >},p€Q, r : Q-S^Q}. 
Given a run p and an acceptance condition a, we assign truth values as follows: 

— Inf{S) is true iff p visits (some state of) S infinitely often, 

— Fin{S) is true iff p visits (all states of) S finitely often, 

— MP^^(r) is true iff lrext(?'(/9[0])r(p[l]) ■■■)\>ip. 

The run p satisfies a if this truth-assignment makes a true. An automaton A is 
an LTS with an acceptance condition a. The language of A, denoted by L(A), 
is the set of all words inducing a run satisfying a. An acceptance condition a is 
a Biichi, generalized Biichi, or co-Buchi acceptance condition if it is of the form 
Inf{S), /\^ Inf (Si), or Fin{S), respectively. Further, a is a generalized Rabin 
mean-payoffs or a generalized Biichi mean-payoff acceptance condition if it is 
in disjunctive normal form, or if it is a conjunction not containing any Fin{S), 
respectively. For each acceptance condition we define a corresponding automaton, 
e.g. deterministic generalized Rabin mean-payoff automaton (DGRMA). 

3 Model-checking algorithm 

In this section, we state the problem of model checking MDPs against fLTL^cu 
specifications and provide a solution. As a black-box we use two novel routines 
described in detail in the following two sections. All proofs are in the appendix. 

Given an MDP M and a valuation n : S ^ 2 "^p of its states, we say that its run 
u) = soaosioi • ■ • satisfies p, written u \= p,il :/(so)j^(si) ■ • ■ \= p. We use '¥'^[p\ 
as a shorthand for the probability of all runs satisfying p, i.e. Pm[{^ I ^ N f }\- 
This paper is concerned with the following task: 
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Controller synthesis problem: Given an MDP with a valuation, an 
fLTL\Qu formula ip and x G [0,1], decide whether > x for some strat¬ 

egy a, and if so, construct such a witness strategy. 

The following is the main result of the paper. 

Theorem 1. The controller synthesis problem for MDPs and /TT'T\gu is de¬ 
cidable and the witness strategy can be constructed in doubly exponential time. 

In this section, we present an algorithm for Theorem [T] The skeleton of our 
algorithm is the same as for the standard model-checking algorithm for MDPs 
against LTL. It proceeds in three steps. Given an MDP M and a formula (/j, 

1 . compute a deterministic automaton A such that L(M) = 

2. compute the product MDP M x M, 

3. analyse the product MDP M x M. 

In the following, we concretize these three steps to fit our setting. 

1. Deterministic automaton For ordinary LTL, usually a Rabin automaton 
or a generalized Rabin automaton is constructed [26122117123) . Since in our set¬ 
ting, along with w-regular language the specification also includes quantitative 
constraints over runs, we generate a DGRMA. The next theorem is the first 
black box, detailed in Section H) 

Theorem 2. For any /LT'T\gu formula, there is a DGRMA A, constructible 
in doubly exponential time, such that L[A) = L{p), and the acceptance condition 
is of exponential size. 

2. Product Computing the synchronous parallel product of the MDP M = 
{S, A, Act, A, s) with valuation v : S ^ and the LTS {Q,i,S) over 2^p 
underlying A is rather straightforward. The product M x A is again an MDP 
{S X Q,Ax Q, Act', A!, (s, 9 )) wher^l Act'{{s, q)) = Act{s) x { 9 }, q = 5{i, r'is)), 
and Z\'((a, q)) ((s, g)) is equal to A{a){s) if S{q,v{s)) = q, and to 0 otherwise. 
We lift acceptance conditions Acc of M to M x A: a run of M x M satisfies Acc 
if its projection to the component of the automata states satisfies Accll 

3. Product aualysis The MDP M x A is solved with respect to Acc, i.e., a 
strategy in M x M is found that maximizes the probability of satisfying Acc. Such 
a strategy then induces a (history-dependent) strategy on M in a straightforward 
manner. Observe that for DGRMA, it is sufficient to consider the setting with 

k 

Acc = \l {Fin{Fi) A Acc'fj (1) 

i=l 

In order to guarantee that each action is enabled in at most one state, we have a 
copy of each original action for each state of the automaton. 

® Technically, the projection should be preceded by i to get a run of the automaton, 
but the acceptance does not depend on any finite prefix of the sequence of states. 
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where Acc'^ is a conjunction of several Inf and MP (in contrast with a Rabin 
condition used for ordinary LTL where Acc'^ is simply of the form Inf{Ii)). Indeed, 
one can replace each /\^ Fin{Fj) by Fin{{Jj Fj) to obtain the desired form, since 
avoiding several sets is equivalent to avoiding their union. 

For a condition of the form o, the solution is obtained as follows: 

1. For z = 1, 2,..., fc: 

(a) Remove the set of states Fi from the MDP. 

(b) Compute the MFC decomposition. 

(c) Mark each MFC C as winning iff AcceptingMFC(C, Acc[) returns Yes. 

(d) Let Wi be the componentwise union of winning MFCs above. 

2. Let W be the componentwise union of all Wz for 1 < z < A:. 

3. Return the maximal probability to reach the set W in the MDP. 

The procedure AcceptingMEC(C', Acc^) is the second black box used in our 
algorithm, detailed in Section [S] It decides, whether the maximum probability 
of satisfying Acc( in C is 1 (return Yes), or 0 (return No). 

Theorem 3. For a strongly connected MDP M and a generalized Biichi mean- 
payoff acceptance condition Acc, the maximal probability to satisfy Acc is either 
1 or 0, and is the same for all initial states. Moreover, there is a polynomial-time 
algorithm that computes this probability, and also outputs a witnessing strategy 
if the probability is 1. 

The procedure is rather complex in our case, as opposed to standard cases such 
as Rabin condition, where a MFC is accepting for Acc) = Inf{Ii) if its states 
intersect A; or a generalized Rabin condition [T2] , where a MFC is accepting for 
Acc) = AjLi if its states intersect with each , for j = 1, 2 ,..., A- 

Finishing the proof of Theorem [T] Note that for MDPs that are not strongly 
connected, the maximum probability might not be in {0,1}. Therefore, the prob¬ 
lem is decomposed into a qualitative satisfaction problem in step l.(c) and a 
quantitative reachability problem in step 3. Consequently, the proof of correct¬ 
ness is the same as the proofs for LTL via Rabin automata [2] and generalized 
Rabin automata [12]. The complexity follows from Theorem [2] and |3| Finally, the 
overall witness strategy first reaches the winning MFCs and if they are reached 
it switches to the witness strategies from Theorem [3| 

Remark 1. We remark that by a simple modification of the product construction 
above and of the proof of Theorem [3] we obtain an algorithm synthesising a 
strategy achieving a given bound w.r.t. multiple mean-payoff objectives (with a 
combination of superior and inferior limits) and (generalized) Rabin acceptance 
condition for general (not necessarily strongly connected) MDP. 

4 Automata characterization of fLTL\GU 

In this section, we prove Theorem [2] We give an algorithm for translating a 
given fLTL^GU formula ip into a deterministic generalized Rabin mean-payoff 
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automaton A that recognizes words satisfying tp. For the rest of the section, let 
ip be an fLTL^cu formula. Further, F, G, G^, and sf denote the set of F-, G-, 
and non-Boolean subformulas of (p, respectively. 

In order to obtain an automaton for the formula, we first need to give a 
more operational view on fLTL. To this end, we use expansions of the formulae 
in a very similar way as they are used, for instance, in tableaux techniques 
for LTL translation to automata, or for deciding LTL satisfiability. We define 
a symbolic one-step unfolding (expansion) Unf of a formula inductively by the 
rules below. Further, for a valuation v C Ap, we define the “next step under 
i^”-operator. This operator (1) substitutes unguarded atomic propositions for 
their truth values, and (2) peels off the outer X-operator whenever it is present. 
Formally, we define 


Unf(7/>i A ^ 2 ) = Unf('i/;i) A Unf('i/;2) 

Unf(^i V ^ 2 ) = Unf('i/;i) V Unf('i/; 2 ) 

Unf(FV’i) = Unf(V'i) VXFV-i 
Unf(GV’i) = Unf(V'i) A XGV’i 
Unf(V^iUV’2) = Unf(V^2)v(Unf(V’i)AX(V^iUV'2)) 

Unf(Grx?^i)=ttAXGrx?^i 

Unf(7/>) = ^ for any other ip 


{ipl A 1p2)M = V’lM A 1p2[l^] 
{ipi V ip2)W] = V’lH V V’2[H 


-<a[iy] = 


ii a G 1 ^ 
\i a 

if a € 
if a ^ 


= Ipl 

ipli/] = Ip for any other ip 

Note that after unfolding, a formula becomes a positive Boolean combination 
over literals (atomic propositions and their negations) and X-formulae. The re¬ 
sulting formula is LTL-equivalent to the original formula. The formulae of the 
form G^^'i/j have “dummy” unfolding; they are dealt with in a special way later. 
Combined with unfolding, the “next step”-operator then preserves and reflects 
satisfaction on the given word: 


Lemma 1. For every word w and fLTL\(^jj formula (p, we have w \= p if and 
only if \= (Unf((/?))[ri;[0]]. 


The construction of A proceeds in several steps. We first construct a “master” 
transition system, which monitors the formula and transforms it in each step to 
always keep exactly the formula that needs to be satisfied at the moment. How¬ 
ever, this can only deal with properties whose satisfaction has a finite witness, 
e.g. Fa. Therefore we construct a set of “slave” automata, which check whether 
“inflnitary” properties (with no finite witness), e.g., FGa, hold or not. They pass 
this information to the master, who decides on acceptance of the word. 



4.1 Construction of master transition system At 

We define a LTS A4 = (Q,(p,S^) over 2^p by letting Q be the set of positive 
Boolean function^ over sf , by letting ip be the initial state, and by letting the 
transition function for every v C Ap and ip € Q, contain ip —^ (Unf(^))[i/]. 
The master automaton keeps the property that is still required up to date: 

Lemma 2 (Local (finitary) correctness of master LTS). Let w be a word 
and A4(w) = ip^ipi--- the corresponding run. Then for all n G N, we have 
w \= (f if and only if u;" ^ ipn. 


Example 1. The formula ip = a A X(6Ua) yields a master LTS depicted below. 


>[aAX(bUa)]- 


{a}, {a, 6} 


0 . 





One can observe that for an fLTL formula p with no G- and G^^-operators, 
we have w \= p Hi the state tt is reached while reading w. However, for formulae 
with G-operators (and thus without finite witnesses in general), this claim no 
longer holds. To check such behaviour we construct auxiliary “slave” automata. 


4.2 Construction of slave transition systems S{^) 

We define a LTS S{f) = {Q, 6^) over with the same state space as A4 and 

the initial state f G Q. Furthermore, we call a state ip a sink, written ip G Sink, 
iff for all v C Ap we have ip[v\ = ip. Finally, the transition relation 6^, for every 
v C Ap and ip € Q \ Sink, contains ipi — ^ ipW]- 


Example 2. The slave LTS for the formula f = aV&VX(6 AGFa) has a structure 
depicted in the following diagram: 


a V 6 V X(& A GFa) 


{a}, {6}, {a, 6} 


L 


>[bA (GFa) 


0.W 


{b}, {a,b} 


GFa 


Note that we do not unfold any inner F- and G-formulae. Observe that if we start 
reading w at the fth position and end up in tt, we have ic* |= f. Similarly, if we 
end up in ff we have ru* ^ This way we can monitor for which position ^ holds 
and will be able to determine if it holds, for instance, infinitely often. But what 
about when we end up in GFa? Intuitively, this state is accepting or rejecting 

® We use Boolean functions, i.e. classes of propositionally equivalent formulae, to ob¬ 
tain a finite state space. To avoid clutter, when referring to such a Boolean function, 
we use some formula representing the respective equivalence class. The choice of the 
representing formula is not relevant since, for all operations we use, the propositional 
equivalence is a congruence, see Appendix lAl Note that, in particular, tt,ff G Q. 
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depending on whether GFa holds or not. Since this cannot be checked in finite 
time, we delegate this task to yet another slave, now responsible for GFa. Thus 
instead of deciding whether GFa holds, we may use it as an assumption in the 
automaton for ^ and let the automaton for GF a check whether the assumption 
turns out correct. 

Let TZec := F U G U This is the set of subformulas that are potentially 
difficult to check in finite time. Subsets of TZec can be used as assumptions to 
prove other assumptions and in the end also the acceptance. Given a set of 
formulae 'f' and a formula ip, we say that ']/ (propositionally) proves ip, written 
'P\-Ip, ii Ip can be deduced from formulae in S' using only propositional reasoning 
(for a formal definition see Appendix [5| . So, for instance, {GFa} hGFa V Gb, 
but GFa /-Fa. 

The following is the ideal assumption set we would like our automaton to 
identify. For a fixed word w, we denote by TZ{w) the set 


(FC e F I n; h GF/} U {G^ £ G | re h FG^j U {G'^,/ G G^ | re h 


of formulae in TZec eventually always satisfied on w. The slave LTS is useful 
for recognizing whether its respective formula / holds infinitely often, almost 
always, or with the given frequency. Intuitively, it reduces this problem for a 
given formula to the problems for its subformulas in TZec: 

Lemma 3 (Correctness of slave LTS). Let us fix / G sf and a word w. For 
any TZ G TZec, we denote by SatiTZ) the set {z G N | 3j > z : 7^h5(^))}. 
Then for any TZ,TZ C TZec such that TZ C TZ{w) C TZ, we have 

Sat(TZ) is infinite => w ^ GF^ Sat{TZ) is infinite (2) 



Before we put the slaves together to determine TZ{w), we define slave automata. 
In order to express the constraints from Lemma [3] as acceptance conditions, we 
need to transform the underlying LTS. Intuitively, we replace quantification over 
various starting positions for runs by a subset construction. This means that in 
each step we put a token to the initial state and move all previously present 
tokens to their successor states. 


Biichi For a formula F^ G F, its slave LTS S{/) = {Q, /, 6^), and TZ C TZec, we 
define a Biichi automaton 5 gf(C) ’^) = (2*^, (Cl) over 2^p by setting 


W — {d^(ip, v) \ Ip gF \ Sink} U {^} for every v C Ap 


and the Biichi acceptance condition Inf{{'F C Q \ 3ip G F H Sink : TZhip}). 

In other words, the automaton accepts if infinitely often a token ends up in 
an accepting sink, i.e., element of Sink that is provable from TZ. For Example [3J 
depending on whether we assume GFa G TZ or not, the accepting sinks are tt 
and GFa, or only tt, respectively. 
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Co-Biichi For a formula G G, its slave LTS S{^) = {Q, S^) and TZ C TZec, 

we define a co-Biichi automaton 5 fg(^) TZ) = {2^, {■C}, <5) over 2^P with the same 
LTS as above. It differs from the Biichi automaton only by having a co-Biichi 
acceptance condition Fin{{'P C Q \ Bij; £ F H Sink ; 72.!/'(/'})• 

Mean-payofF For a formula G^j^ G its slave LTS S{^) = and 

TZ C TZec we define a mean-payoff automaton iSg^p(^, 72) = (IQI*^, Ij, (5) over 
2"^P so that for every v C Ap, we have / /' where 

/'(■*/'') = fifj). 

Intuitively, we always count the number of tokens in each state. When a step is 
taken, all tokens moving to a state are summed up and, moreover, one token is 
added to the initial state. Since the slave LTS is acyclic the number of tokens in 
each state is bounded. 

Finally, the acceptance condition is MF^j(r(72)) where the function r{TZ) 
assigns to every state / the reward: 

Y 

'ip£S\nk,TZ h 'll’ 

Each state thus has a reward that is the number of tokens in accepting sinks. 
Note that each token either causes a reward 1 once per its life-time when it 
reaches an accepting sink, or never causes any reward in the case when it never 
reaches any accepting state. 

Lemma 4 (Gorrectness of slave automata). Let £, G sf, w, andTZ,TZ C TZec 
be such that TZ C TZ{w) C TZ. Then 

w G L(5 gf(C)^)) ui 1= GF^ => ui G £(5 gf(Ci'^)) (5) 

w £ L{Sfg{^,TZ)) whFGC w £ L{SFG{f,.TZ)) (6) 

w £ L{S^^j.{^,TZ)) ^ G^x^e ^ w£ (7) 

4.3 Product of slave automata 

Observe that the LTS of slave automata never depend on the assumptions TZ. Let 
Si,... ,Sn be the LTS of automata for elements of TZec = {^i,..., Further, 
given TZ C TZec, let AcCi{TZ) be the acceptance condition for the slave automaton 
for with assumptions TZ. 

We define 'P to be the LTS product 5i x • • • XiS„. The slaves run independently 
in parallel. For TZ C TZec, we define the acceptance condition for the produc10 

^cc(72) = /y ^cci(72) 

Uen 

^ An acceptance condition of an automaton is defined to hold on a run of the automata 
product if it holds on the projection of the rnn to this antomaton. We can still write 
this as a standard acceptance condition. Indeed, for instance, a Biichi condition for 
the first automaton given by F C Q is a Biichi condition on the product given by 
{(<?!, <72,..., (7n) \qi£F,q2,...,qn£ Q}. 
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and V{'R) denotes the LTS V endowed with the acceptance condition AccijV). 
Note that Acc{TV} checks that IZ is satisfied when each slave assumes TZ. 

Lemma 5 (Correctness of slave product). For w and TZ C TZec, we have 

(soundness) whenever w G L{V{TZ)) then TZ C TZ{w); 

(completeness) w G L{'P{TZ{w))). 

Intuitively, soundness means that whatever set of assumptions we prove with T’ 
it is also satisfied on the word. Note that the first line can be written as 

weL{V{TZ)) ^ zrh A A FGCA A 

Fie-R gcgR 

Completeness means that for every word the set of all satisfied assumptions can 
be proven by the automaton. 


4.4 The final automaton: product of slaves and master 

Finally, we define the generalized Rabin mean-payoff automaton A to have the 
LTS M xV and the acceptance condition Vrcrbc Accm{T^) A AcciJZ) where 


Accm{TZ) = Fin (^«)«eRec) 


7^U y Tr^[{TZec\TZ)/W\^i,]) 

G{eR 


eventually prohibits states where the current formula of the master ■0 is not 
proved by the assumptions and by all tokens of the slaves for G TZ. Here 
<F[X/ff] denotes the set of formulae of Tr where each element of X in the Boolean 
combination is replaced by ff. For instance, {a V Fa}[{a}/ff] = ff V Fa = Fa. 
(For formal definition, see Appendix]^) We illustrate how the information from 
the slaves in this form helps to decide whether the master formula holds or not. 

Example 3. Consider ip = G(Xa V GX6), and its respective master transition 
system as depicted below: 

{a},{a,6}p^ {6}, {a, 6}^ 

0 ,{a},{&},{a,6} ^- Li -^ {6} ^- Li- ^ 

- A (a V (6 A GX6)))- Ap A (fe A GXfo)) 


{fe}. 

Assume we enter the second state and stay there forever, e.g., under words {a}"^ 
or {a, b}‘^. How do we show that i^A (aV (6AGX6)) holds? For the first conjunct, 
we obviously have TZ\-ip for all TZ containing p. However, the second conjunct 
is more difficult to prove. 

One option is that we have GX& G TZ and want to prove the second disjunct. 
To this end, we also need to prove b. We can see that if GX6 holds then in its 
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Xi,a 

= 1 

for all 1 < i < n 

(8) 

V Xi,aA{a){s) 

= Xi^a 

' ^ aGAct{s) 

for all s £ S' and l<i<n 

(9) 

. Xi,a ■ rj(s) 

' Act(s) 

tXl Vj 

for all l<j<m and l<i<n 

(10) 

‘ ^ sG S .aG Act{ 

to Ui 

for all l<i<n 

(11) 


Fig. 1: Linear constraints L of Proposition [3] 

slave for X6, there is always a token in the state b, which is eventually always 
guaranteed to hold. This illustrates why we need the tokens of the G-slaves for 
proving the master formula. 

The other option is that GX& is not in TZ, and so we need to prove the first 
disjunct. However, from the slave for G(XaVGX&) we eventually always get only 
the tokens Xa V GX6, a V GX&, and tt. None of them can prove a V (& A GX&). 
However, since the slave does not rely on the assumption GX6, we may safely 
assume it not to hold here. Therefore, we can substitute fF for GX6 and after the 
substitution the tokens turn into Xa, a, and tt. The second one is then trivially 
sufficient to prove the first disjunct. 

Proposition 1 (Soundness). If w G L{A), then w\= (f. 

The key proof idea is that for the slaves of G-formulae in 72., all the tokens 
eventually always hold true. Since also the assumptions hold true so does the 
conclusion fj. By Lemma [21 (p holds true, too. 

Proposition 2 (Completeness). If w \= p>, then w G £(A). 

The key idea is that subformulas generated in the master from G-formulae closely 
correspond to their slaves’ tokens. Further, observe that for an F-formula y, its 
unfolding is a disjunction of y and other formulae. Therefore, it is sufficient to 
prove y, which can be done directly from 72. Similarly, for G^^-formula y, its 
unfolding is just y and is thus also provable directly from 72. 

Complexity Since the number of Boolean functions over a set of size n is 2^ , 
the size of each automaton is bounded by 2^' ', i.e., doubly exponential in the 
length of the formula. Their product is thus still doubly exponential. Finally, 
the acceptance condition is polynomial for each fixed 72 C 72ec. Since the whole 
condition is a disjunction over all possible values of 72, it is exponential in the 
size of the formula, which finishes the proof of Theorem [2] 

5 Verifying strongly connected MDPs against generalized 
Biichi mean-payoff automata 

Theorem |3| can be obtained from the following proposition. 
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Proposition 3. Let M = {S, A, Act, 5, s) be a strongly connected MDP, and Ace 
an acceptance condition over S given by: 

The constraints from Figure [7] have a non-negative solution if and only if there 
is a strategy a and a set of runs R of non-zero probability such that Acc holds 
true on all uj £ R. Moreover, a and R can be chosen so that R has probability 1. 

Intuitively, variables xt^a describe the frequencies of using action a. Equation 
is Kirchhof’s law of flow. Equation (nni says the inferior limits must be satisfied 
by all flows, while Equation (El) says that the ith limit superior has its own 
dedicated ith flow. Note that L does not dependent on the initial state s. 

Proof (Sketch). Existing results for multi-objective mean payoff MDPs would 
only allow to establish the proposition in absence of supremum limits, and so 
we need to extend and combine results of several works to prove the proposition. 
In the direction =>, na Corollary 12] gives a strategy ai for every i such that 
for almost every run soaosioi... we have hinf((llat=o)t^o) “ Xi,a, and in fact 
the corresponding limit exists. Hence, for the number p = X^seS a^Act{s )' 
Xi^a the predicates MP^{r) and almost surely holds, for any reward 

function r. Hence, our constraints ensure that ai satisfies MP^(^ (j-j) for all j, 
and MP^p‘(gi). Moreover, ai is guaranteed to visit every state of M infinitely 
often almost surely. The strategy a is then constructed to take these strategies 
ai,l < i < n in turn and mimic each one of them for longer and longer periods. 

For the direction <^, we combine the ideas of mm and select solutions 
to Xi^a from “frequencies” of actions under the strategy a. 

6 Conclusions 

We have given an algorithm for computing the optimal probability of satisfying 
an fLTL^Qu formula in an MDP. The proof relies on a decomposition of the 
formula into master and slave automata, and on solving a mean-payoff problem 
in a product MDP. The obvious next step is to extend the algorithm so that it 
can handle arbitrary formulae of fLTL. This appears to be a major task, since 
our present construction relies on acyclicity of slave LTS, a property which is 
not satisfied for unrestricted formulae m- Indeed, since C^P-slaves count the 
number of tokens in each state, this property ensures a bounded number of 
tokens and thus finiteness of the slave automata. 
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A Propositional reasoning 

Intuitively, given a set of formulae <P and a formula '4’t we say that ^ proposition- 
ally proves Ip ii 'tp can be deduced from formulae in (p using only propositional 
reasoning. So, for instance. Go propositionally implies Go V Gb, but Ga does 
not propositionally imply Fa. 

Definition 1 (Propositional implication and equivalence). A formula of 
fLTL is non-Boolean if it is not a conjunction or a disjunction (i.e., if the root 
of its syntax tree is not A orV). The set of non-Boolean formulae of fLTL over 
Ap is denoted by NB{Ap). A propositional assignment, or just an assignment, 
is a mapping Ass: NB(Ap) —> {0,1}. Given ip € NB(Ap), we write p 

iff Ass(p) = 1, and extend the relation ^p to arbitrary formulae by: 

\=p p A-ip iff Ass \=p p and \=p 
^ss |=p pW ip iff Ass \=p p or |=p ip 
We say that a set <P of fLTL formulae propositionally proves an fLTL formula 
Ip, written <P\-ip, if for every assignment ^ss, ^ss ^p /\^ implies ^ss ^p ip. 

Finally, fLTL formulae p and ip are propositionally equivalent, denoted by 
p =p Ip, if {p} ^p Ip and {ip} ^p p. We denote by {p\p the equivalence class 
of p under the equivalence relation =p. 

Observe that p =p ip implies that p and ip are equivalent also as fLTL 
formulae, i.e., for all words w, we have w \= p lA w \= ip. Using the same 
reasoning, 

w \= l\^T> with <L>\-ip imply w \= ip. (12) 

Definition 2 (Propositional substitution). Let ip, x be fLTL formulae and 
W C NB(Ap). The formula ip^/x\p is inductively defined as follows: 

- If Ip = ipi Aip 2 thenipfT/x\p = 'pii\T/x\p I^'>p 2 \f['/x\p. 

- If P’ = P ’2 then ip[^/x]p = V’il^/x]p V V’ 2 [^'/x]p- 

- If Ip is a non-Boolean formula and ip gF then ip[F/x]p = X» eke ip[F/x]p = 
Ip. 

The following lemma allows us to work with formulae as Boolean functions 
over NB{Ap), i.e., as representatives of their propositional equivalence classes. 

Lemma A For every formula p and every letter v G 2^^, if pi =p p 2 then 
Unf((/?i))[p] =p Unf((/? 2 ))[p]- 

Proof. Observe that every formula (/? is a positive Boolean combination (i.e., 
built from conjunctions and disjunctions) of non-Boolean formulae. Since Unf 
and (OH both distribute over A and V, the formula Unf((/?)[p] is obtained by 
applying a simultaneous substitution to the non-Boolean formulae. (For example, 
a non-Boolean formula Gip is substituted by Unf (O’) [p] A Gip.) Let V3[<S'] be the 
result of the substitution. 

Consider two equivalent formulae pi =p p 2 . Since we apply the same sub¬ 
stitution to both sides, the substitution lemma of propositional logic guarantees 
VSlH] =P P2[S]. 
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B Proofs of Section 0] 


Lemma [TJ For every word w and fLTL formula, we have w \= (p ^ 

(UnfM)[w[0]]. 

Proof. Denote w = vv where v C Ap. We proceed by a straightforward structural 
induction on p. We focus on three representative cases. 


— p = a. Then 


— p = Ftp. Then 


lyv \= a 

a € V (semantics of LTL) 

Unf(a)[:/] = tt (def. of Unf and [ly]) 
V \= Unf(a)[z/] (semantics of LTL) 


i/v 1 = F'^ 

lyv \= (XFV’) V V' {F-ip = X.FiP V ip) 

V \= Fip or lyv \= Ip (semantics of LTL) 

V \= Fip or 1= \Jn^{ip)[v\ (ind. hyp.) 

V \= Fip V Unf(-0)[i/] (semantics of LTL) 

V ^ Unf(F'!/')[!^] (def. of Unf) 


- F = V'- Then 


.v^G^^iP 

liminf 

i^oo I \ ' ^ 


i-2 

'^vi\=ip 

3=0 


i-2 


(semantics of LTL) 


lim + liminf - 

i—>.oo ^ ^^ J 


3=0 


1 ! \ 

0 + lim inf — > |= 

2^00 i ' 




^ h Gf^ip 
V h Unf(G^fV')H 


(semantics of LTL) 
(def. of Unf) 


Lemma Let w be a word and M{w) = popi ■ ■ ■ the corresponding run. Then 
for all n G N, we have w \= p if and only if w" |= pn. 
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Proof. We proceed by induction on n. For n = 0, we conclude by ipo = if. Let 
now n > 1 and denote w = uv v where v C AP and v = vf^. Then we have 


V ^ \}n^{fn-i)[v] 

fn-1 

UVV \= f 


(def. of 6^) 
(Lemma [T]) 
(ind. hyp.) 


□ 


Definition 3. The threshold T{w) of a word w is the smallest T G N such that 
for all t > T 

— for all ip G 7?.(u>), we have w* ^ 

— for all Ip € TZec \ TZ{w), we have %p. 

Then we have |= p for every p € 7Z(w) (all G-formulae that will ever 

hold do hold already) and ^ p for every p £ TZec \ TZ{w) (none of the 

F-formulae that hold only finitely often holds any more). 

Lemma B For every word w and t > T{w), we have that w* \= f iff 3t' : 

7^(u;)nsf (Ob 5(0 («;“'). 

Proof. By similar arguments as in Lemma [21 we get that for the run of the slave 
S{f){w*) = 00+1 ■ • • we have lu* |= ^ w* |= O'- Indeed, not unfolding 

elements of TZec is here equivalent to not unfolding them since for every f) £ TZec 
we have ?c“ \= tp iSw'^ 0 u > T. Moreover, when reaching the sink at 

time t', we know that 0' is a positive Boolean combination over TZec{w) fl sf(0- 
Therefore, lu* \= ^ w* 0 0' TZ{w) |= 0' 7^(w) n sf(0 0 0'- 

□ 


Lemma |3l Let us fix f £ sf and a word w. For any TZ £ TZec, we denote by 
Sat{TZ) the set {i S N | f : 72.1-5(0(11’^)}. Then for any TZ,TZ C TZec such 

Sat{TZ) is infinite (13) 

N \ Sat{TZ) is finite (14) 

hext((ls„t('^)(i))^o) ixip (15) 

Moreover, the result holds also for TZ n sf(0 0 TZ{w) fl sf(0 C 72 fl sf(0- 

Proof. For (IT^ . let first Sat(JZ) be infinite. Then also Sat'ifjV) := {n £ Sat{TZ) \ 
n > T{w)} is infinite. Therefore, infinitely many positions i oi w satisfy 3j > 
i : TZ\- S{^){w'^^). Observe that elements of TZec are never under the scope of 
negation in Q, hence h is monotonic w.r.t. adding assumptions from TZec. Thus 

® This condition is actually non-trivial only for G-formulae, other formulae of 7Z{w) 
hold at all positions. 


that TZ C TZ{w) C TZ, we have 

SatffZ) is infinite => w ^ GF^ 
N \ SatffZ) is finite => w ^ FG^ 

lrext((lsat(K)(0)“o) ^ W h 
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also infinitely many positions i oi w satisfy 3j > i : 'R.{w)\- and by 

Lemma iBl also satisfy 

Let now w \= GF^. Then I := {i € N \ i > T{w) and w® \= is infinite 
and by the lemma there are infinitely many positions i oi w satisfying Bj > i : 
TZ\- By the monotonicity of h above we can replace TZ by TZ. 

Moreover, if we only assume ^nsf(^) C TZ{w) nsf(^) C 7?.nsf(^) both state¬ 
ments remain valid. Indeed, for every set TZ of formulae and formula reachable 
from 72.1-^ iff 7?. fl sf(^) since the only non-Boolean formulae produced by 
^[•] are subformulas of 

For (fT4)) . the argumentation is the same, replacing “infinite” and “infinitely 
many” by “co-finite” and “almost all”. For (flSl) , the sequences can only differ in 
a finite prefix. Moreover, if we only assume ^nsf(^) C TZ{w) nsf(^) C 72.nsf(^), 
apart from the finite prefix the sequence sat{Ti){'i) is pointwise less or equal to 
which is again pointwise less or equal to 

Lemma |4l Let ^ S sf, w, and ]Z,TZ C TZec be such that TZ C TZ{w) C TZ. Then 

weliScFitm) w\=GF^ ^ w G LiSGF{^,TZ)) (16) 

w e LiSFGi^TZ)) whFGC ^ w € L{Sfg{^,TZ)) (17) 

w e L{Sg^j{^,TZ)) ^ G^x^C ^ w€ i(5G-p(C,^)) (18) 

Moreover, the result holds also for TZ fl sf(^) C TZ{w) fl sf(^) C 7^ n sf(^). 

Proof. Due to Lemma [3l it suffices to prove for the given f and w and for any 
TZ that 


Sat{TZ) is infinite <^= 

w £ L{Sgf{£,,TZ)) 

(19) 

N \ Sat (TZ) is finite <^= 

w G L{SFGi£,,TZ)) 

(20) 

lrext((ll5at(7?,)(0)“o) <(= 

u> G L(5 g^p(^, 7?.)) 

(21) 


For (jl9ll . we must prove that there are inhnitely many positions from which 
the run ends in an accepting sink iff there are infinitely many positions with a 
token in an accepting sink. To this end, observe that to each position j with 
a token in an accepting sink q (i.e., TZLq.) we can assign a set Endln{j, q) of 
positions i such that 5 gf(C)('*^*'^ ) = 9 - Oii Ih® hand, each i is exactly in one 
Endln{j, q) since the slave transition systems are acyclic and each path inevitably 
ends in a sink. On the other hand, each Endln{j, q) is hnite, again due to the 
acyclicity. Consequently, Sat (TZ) is infinite iff q Endln{j,q) is infinite iff the 
number of non-empty Endln{j,q) is infinite iff 5 gf accepts. 

For ([20]), the argument is analogous, but we have to consider Endln{j,q) for 
rejecting sinks q, i.e., TZ /r q. Then SatlJZ) is co-finite iff q Endln(j,q) is 
finite iff the number of non-empty Endln{j, q) is finite iff iSfg accepts. 

For (EU, observe that in iSqxp the precise number of tokens is preserved in 
each state at every point of time. Therefore, each successful run corresponds 
exactly to one 1 in the total reward. In order to prove that both sequence have 
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the same liminf / limsup, we need to prove that the length of each run (difference 
between the element’s positions in the two sequences) is bounded. This follows 
by acyclicity of the automaton. □ 


Lemma For w and TZ C TZec, we have 

(soundness) whenever w € L{V{TZ)) then TZ C TZ{w) and hence 

w\= /\ GF^A /\ FG^A /\ 

(completeness) w € L{'P{TZ{w))). 


Proof. As to soundness, let w G L(S(TZ)). Consider the dag on TZ given by an 
edge (x,xO if x' ^ ^Hx) \ {x}- We prove the right-hand side of the implication 
for each formula ^ € 72. by induction on the distance d to the leaf in the dag. 

Let d = 0 and consider x = F^; the other cases are analogous. Then f 
does not contain any subformula from TZ. Therefore, not only w € L(iSgf(^, 72), 
but also w G L(5gf(^,0)- Since 0 C TZ{w), Lemma 0] (part “Moreover”) yields 
w h GF^. 

Let d > 0 and x = F^; the other cases are again analogous. We have not only 
w G L(5gf(^,72), but also w G L{Sgf{£,,TZ n sf(^)). By induction hypothesis, 
w \= 72nsf(^). Therefore, 72nsf(^) C TZ{w) and thus Lemma |4] yields w |= GF^. 

As to completeness, we prove that w G L(5, 5gf(72(ic))) for F^ G TZ{w); 
the proof for other types of automata is analogous. Since F^ G TZ{w) we have 
w \= GF^. By Lemma|4]we have w G lj{SGF{i,TZ(w))). □ 

We call the left-hand-side of h of the acceptance condition “extended assump¬ 
tions” since it is a conjunction of assumptions 72 extended by 'Z'^[(72ec\72)/fF] for 
each G^ G TZ. We prove the extended assumptions hold at almost all positions: 

Lemma G For every word w accepted with respect to TZ, and for every formula 
Gf G TZ, and for all t > T(w) and for all t > t such that if := ScFiOiw*'^) is 
defined, we have that ru” ^ -iplTZec \ 72/ff]. 

Proof. Any tokens born at time t > T{w) will end up at some time d > t in an 
accepting sink ss. Since \= TZ{w) and TZ{w) h s^, we have |= s^. Since also 
w'^ \= TZ{w) for all r G [7,d], we obtain also ic” \= ip hy similar argumentation 
as in Lemma IB] 

Moreover, since TZ{w) h st, by propositional calculus TZ{w) h S5[72ec\72(rc)/ff] 
and \= S5[72ec\72(w)/ff], and similarly we obtain w” |= '!/)[72ec\72(w)/ff]. □ 
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Proposition [TJ If w G L{A), then w\= tf. 


Proof. If ly € i^{A) and it accepts by a disjunct in its acceptance conditions 
related to assumptions TZ C TZec, then for almost all positions t when visiting a 
state {fj, (iI'jjjg-R.ec) we have 

7eu y w^[nec\n/s]hfj 

G^GTZ 

and, moreover by Lemma [5] and [Cl we also have 

w*^nu y w^[nec\n/s] 

G^eTi 

yielding together by m 

w* \= tp 

which by Lemma [2] gives w \= ip. □ 


Proposition If w \= ip, then w G L{A). 


Proof. Let w be a word. Then Acc{TZ{w)) is satished by Lemma jS] We show 
that Accm{T^{w)) is satished, too. In other words, we prove that for almost all 
all positions t when visiting a state {ip, (ifjjjgTjec) we have 

TZ{w)U y ^'{[T^.ec \ 7?.(u>)/ff] hi/) 

G^eTliw) 

Since both ip and each element of each are actually Boolean functions, we 
choose formulae that are convenient representations thereof. Namely, we consider 
the formula generated exactly from or ^ using the transition functions 6^ or 
6^, respectively. Therefore, each occurrence of G sf((/j) corresponds after 
reading a hnite word v to some occurrence of ip' G si {ip) where ip' = A /\j 
and = 5^{^,Vi) for some inhx Vi of w, we call such a formula ip' derived 
G -subformula. Similarly, reading v transforms into a derived F-subformula 
Ff V Finally, similarly for G^^,” U-formulae. Note that every derived 

is always of the form G^^^ A /\ tt. 

We consider positions large enough so that 

— they are greater than T{w) + |Q| (here |Q| ensures that tokens born before 
T{w) do not exist any more), and 

— all the satisfied U-formulae have their second argument already satisfied, 
and 

— Ip is a, Boolean combination over derived formulae since all outer literals and 
X-operators have been already removed through repetitive application of [•]. 
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We prove that each derived formula (in ip) that currently holds is also 
provable from the extended assumptions. Since ip holds, this implies that also 
the whole ip is provable from the extended assumptions. We proceed by structural 
induction. 

First, let ip' be a derived G-subformula Gp^A/\^ Since ip' holds, holds, 
we have G^ G TZ{w) and thus TZ{w) h G^. Further, each corresponds to a 
formula ipi either in or a sink, which is accepting since holds, as follows: 
This correspondence mapping is very similar to identity, except for 

— each derived F-formula Fy V Vi X* is mapped to Fy since S{^) does not 
unfold F, and 

— each derived G-formula Gy A f\^Xi is mapped to Gy since <S(^) does not 

unfold G, moreover, each yj again corresponds in the same way to a formula 
in accepting sink of Sgy{T^(w), x) by the induction hypothesis. 

If we could replace each derived formula in by its simple image in the correspon¬ 
dence mapping, we would have ipi h (and since ipi is provable from assumptions 
- either a token or an accepting sink - we could conclude). Therefore it remains 
to prove all the derived formulae: 

— Gy A /\ - Xi that holds can be proved by induction hypothesis, 

— Gy A /\j Xi that does not hold is proved from Gy[7^ec \ 'lZ{w)/S\ = fF 

— Fy V /\j Xi that holds is proved from Fy G 'lZ{w) 

— Fy V /\j Xi that does not hold is proved from Fy[7^ec \ 'lZ{w)/S\ = IF 

Second, ip' = G^^^ A /\ tt is proved directly from TZ{w). 

Third, let (i) ip' be a derived F-subformula F^ V \J ■ such that F^ holds. 
Then F^ G TZ{w) and thus TZ{w) h F^. 

Finally, let ip' be a derived F-subformula F^ V Vi such that F^ does not 
hold (i.e., some of the ^i’s hold), or a derived U-subformula, where thus one 
of the disjuncts not containing this until holds (since all satisfied untils have 
their second argument already satisfied). Then we conclude by the induction 
hypothesis. □ 

C Probability space of Markov chain 

For a Markov chain N = {L,P,£) we define the probability space {Run, T,V) 
where 

— Run contains all runs initiated in £, i.e. all infinite sequences £o£i... satisfy¬ 
ing £o = £ and P{£i, f^i-i-i) > 0 for all f > 0. 

— J' is the cr-field generated by basic cylinders Cyl{h) := {uj\oj starts with h} 
for all h which are a prefix of an element in Run. 

— P is the unique probability function such that for h = £q£\ ■ ■ .£n we have 

When we say “almost surely” or “almost all runs”, it refers to an event 
happening with probability 1 according to the relevant measure (which is usually 
clear from the context). 
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D Proof of Proposition [3] 

In the rest of this section we prove Proposition [S] To simplify the notation, 
for an action a and reward structure r we will use lrext(«) and Irext(r) for ran¬ 
dom variables that on a run io = sgaoSiai... return lrext(lao=alai=a • ■ •) and 
hext(rsorsj ...), respectively. 

The direction ^ can be proved as follows. For any fixed by m Corollary 
12 ] there is a strategy ai such that lr(a) = Xi^a almost surely, and hence Ui almost 
surely yields reward any reward function r. Moreover, 

visits every state of M infinitely often almost surely. 

We now construct cr inductively as follows. The strategy will keep the current 
“mode”, which is a number from 1 to n, and an unbounded “timer” ranging over 
natural numbers. Suppose we have defined a for history h, but not for any other 
history starting with h. Suppose that in the history before h the strategy a was 
in mode Then in h the mode is incremented by 1 (modulo n), yielding the 
mode and the strategy cr starts playing as cr^/. It does so for steps, yielding 
a history h'. Afterwards, we apply the inductive definition again with h' in place 
of h. 

Lemma 6 . The strategy a satisfies the requirements of Proposition\^ 

Proof (Sketch). Firstly, the generalised Biichi condition is almost surely satis¬ 
fied because it is satisfied under any cr^, and a will eventually mimic cr^ for an 
arbitrary long length. 

Let us continue with the claim for the mean payoffs with supremum limits. 
Fix 1 < i < n. We will show that for every e > 0, almost every run w has a 
prefix sooosiai ■ ■ - sg with 


>Ui-£ 

By properties of (Ji for any s € S there is a number kg^e and a set of runs R 
with PJ’ [T] >1/2 such that for every ... € T and every k' > fcs,e we 

have 

k' ^ 

1=0 ^ 

Let a be the smallest assigned reward, there must be a number such that 

Je ■ (Ui - a) < 2 “^' • e/2 

Intuitively, is chosen so that no matter what the history is in the first steps, 
if the remainder has length at least 2 ^ steps and gives partial average at least 
Ui — e/ 2 , we know that the whole history gives a partial average at least ut — e. 

Now almost every run w has infinitely many prefixes /iq, hi... such in the 
prehx hi, the strategy cr starts mimicking ai for 2l^’l steps. Now consider those 
prehxes hi which have length greater than maxg kg^s and Jg. This ensures that 
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starting with any such prefix with probability at least 1/2 the history h' = 
SQa^s'^a'^ ... in which we end after taking 2^^'^ steps will satisfy 

1 

>u^-e 

j=o 

Using Borel-Cantelli lemma [28] this implies that almost every run has the re¬ 
quired property. 

The proof for mean payoff with inferior limits is analogous, although handling 
limit inferior is more subtle as it requires us to show that from some point on 
the partial average never decreases below a given bound. To give a formal proof, 
we can reuse the construction from [H Proof of Claim 10] applied to strategies 
(Cfc)i<fe<oo where each for k of the form i- j + i is defined to be the strategy 
(Ti. Note that our choice of “lengths” of each mode satisfy Equations (3) and 
(4) from [131 Proof of Claim 10]. Also note that while [T3I Proof of Claim 10] 
requires the frequencies of the actions to converge, in our proof we are only 
concerned about limits inferior of long run rewards, and so the requirements on 
are not convergence of limits, but only that limits inferior converge to the 
required bound. This requirement is clearly satisfied. □ 

Let us now proceed with the direction <^= of the proof or Proposition [Sj Be¬ 
cause no Xi^a and Xi'^a' with i ^ i' occur in the same equation, we can fix 
1 < i < TO and to hnish the proof it suffices to give a solution to Xi^a for all a. 

Similarly to mm where only lim inf was considered, the main idea of the 
proof is to obtain suitable “frequencies” of actions and use these as the solution. 
Nevertheless, the formal approach of mm itself cannot be easily adapted (main 
issue being the use of Patou’s lemma, which for the purpose of limit superior does 
not allow to establish the required inequality). Instead, we use a straightforward 
adaptation of an approach used in [ID]. The statement we require is captured in 
the following lemma. 

Lemma 7. For every run uj = SoagSiai... there is a sequence of numbers 
Ti [w], r 2 [w]) • ■ • such that the number 


fiuia) := lim —--r V la,.=a 

J j=l 

is defined and non-negative for all a G A, and satisfies 

Y.aeA fUa) ■ q^{a) = lrsup(gi)(a;) 

EaGA /‘^(«) • ^ for 1 < j < n 

EaGA/-(a) = 1 

Moreover, for almost all runs uj we have 

^E(a) •<5(a)(s) = fu,{a) 

a^A aGAct{s) 


25 







Proof. Fix w = soaoSifli... We first define a sequence [w], T 2 [w],... to be any 
sequence satisfying 


rpir 1 — lrsup( 9 i)(w) 


Existence of such a sequence follows from the fact that every sequence of real 
numbers has a subsequence which converges to the lim sup of the original se¬ 
quence. 

Further, we define subsequences Tf [w], [w],... for 1 < fc < n where for all 

k the sequence [w], TI" [w],... satisfies 

=lrsupfe)H 


and 


lim 

.^^00 


^ ffH 
Tf[co] ^ 


i=i 


> lri„f(rfe')(w) 


for all k' <k. We define these subsequences inductively. We start with T® [w], [w].. 

T{[a;], T 2 [u;] _Now assuming that ... = 0,1... has been defined, we 

take [cc], T*[u;],... such that 


lim 

I—¥00 


T^[u 


E 

i=i 


rk{aj) 


exists. The existence of such a sequence follows from the fact that every sequence 
of real numbers has a converging subsequence. The required properties then 
follow easily from properties of limits. 

Now assuming an order on actions, di,..., d|^| in A, we define [ct>], T*[w],... 
for 0 < k < 1^1 so that r°[a;], T 2 [w],... is the sequence T”[a;], ..., and 

every Tf[uj],T 2 [u)],... is a subsequence of T 2 ... such that the 

following limit exists 


fcjiak) := lim 


Ttiuj] 


■E 

i=i 


The required properties follow as before. We take ... to be the 

desired sequence Ti [w], r 2 [w],.... 

Now we need to show that satisfies the required properties. Indeed 


^ TfH 

Y. fUa) ■ q^{a) = Y E 

a^A a^A ^ ^ ^ j—^ 


(def. of /^(a)) 
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1 


(property of 1 and the sum) 


= lim 


f->oo 

— hsup(Q2)(^) 
and analogously, for any 1 < i' < n: 


qi[a) 


j=i 


(def. of subsequence 


oeA 


Y /-(«) • («) = I] (“) 

a^A ^ ^ -I j=l 


1 


Also 


= lim , , 

t^co [ cli ] 

> lrinf(»'i')(‘^) 


Y 




fuiia) = lim — rr^ la -a = lim —y^ 1 =1 

^ ’ ^J^ocT^[uj] ^ e^ooT^[oj] ^ 

a^A a^A ^ '• -* j — 1 ^ '• -* j — 1 

To prove the last property in the lemma, we invoke the law of large numbers 
(SLLN) [20]. Given a run w, an action a, a state s and fc > 1, define 

1 a is executed at least k times 
N^’‘^{u}) = and s is visited just after the fc-th execution of a; 

0 otherwise. 

By SLLN and by the fact that in every step the distribution on the next states 
depends just on the chosen action, for almost all runs uj the following limit is 
defined and the equality holds whenever /aj(a) > 0: 

hm fc V ^ ^ <5(a)(s) 

j^co J 

We obtain, for almost every uj = soao^iai... 

Y ■ '^(«)(s) 

Ti[oj] 


aGA 


= y^ lim —p-r y^ la -a • lim - y^ 7V((’®(a;) 
aeA ^ ^ j—1 k—1 


T^M 


~ Y^ S ^aj=a • 


^j = l “j- 


lim 


aeA 


i-^oD TAuj] ^ “ ^->-00 I 

' i=i Z^j=i 


E 


■j—a k—1 


A^j = l aj—a 


= E 1™ rr r 1 E 
^ 1^00 Tiluj] ^ k y / 


aGA 


k^l 


27 












1 


la =a 

^j = l aj—a 


= lim y y 


a^A k—1 
T,M 


= lim —Is =s 
f-»oo Te[uj] ■’ 


T,H 


J j=l aeAct(s) 


= E 

aGAct(s) ^ J j=l 

= y ^(a) 

aGAct(s) 


T,[u 


□ 

We apply Lemma [7] to obtain values fuj for every w. Now it suffices to con¬ 
sider any uj for which /(^ satisfies the last condition of the lemma and which 
also satisfies lri„f(rj[u;]) > Uj for all 1 < j < n and lrsup(( 3 'i[a;]) > vf, by the 
assumptions on a and R such a run must exist. This immediately gives us that 
all the equations from Figured] are satisfied. 
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